{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Safemail-BERT.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true,
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/jason1416/Predicting-Movie-Reviews-with-BERT/blob/master/Safemail_BERT.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "j0a4mTk9o1Qg",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Copyright 2019 Google Inc.\n",
        "\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "\n",
        "#     http://www.apache.org/licenses/LICENSE-2.0\n",
        "\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dCpvgG0vwXAZ",
        "colab_type": "text"
      },
      "source": [
        "#Predicting Movie Review Sentiment with BERT on TF Hub"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xiYrZKaHwV81",
        "colab_type": "text"
      },
      "source": [
        "If you’ve been following Natural Language Processing over the past year, you’ve probably heard of BERT: Bidirectional Encoder Representations from Transformers. It’s a neural network architecture designed by Google researchers that’s totally transformed what’s state-of-the-art for NLP tasks, like text classification, translation, summarization, and question answering.\n",
        "\n",
        "Now that BERT's been added to [TF Hub](https://www.tensorflow.org/hub) as a loadable module, it's easy(ish) to add into existing Tensorflow text pipelines. In an existing pipeline, BERT can replace text embedding layers like ELMO and GloVE. Alternatively, [finetuning](http://wiki.fast.ai/index.php/Fine_tuning) BERT can provide both an accuracy boost and faster training time in many cases.\n",
        "\n",
        "Here, we'll train a model to predict whether an IMDB movie review is positive or negative using BERT in Tensorflow with tf hub. Some code was adapted from [this colab notebook](https://colab.sandbox.google.com/github/tensorflow/tpu/blob/master/tools/colab/bert_finetuning_with_cloud_tpus.ipynb). Let's get started!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hsZvic2YxnTz",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 63
        },
        "outputId": "12eb1fa3-83e9-4b32-cb45-1776093f4c13"
      },
      "source": [
        "from sklearn.model_selection import train_test_split\n",
        "import pandas as pd\n",
        "import tensorflow as tf\n",
        "import tensorflow_hub as hub\n",
        "from datetime import datetime"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<p style=\"color: red;\">\n",
              "The default version of TensorFlow in Colab will soon switch to TensorFlow 2.x.<br>\n",
              "We recommend you <a href=\"https://www.tensorflow.org/guide/migrate\" target=\"_blank\">upgrade</a> now \n",
              "or ensure your notebook will continue to use TensorFlow 1.x via the <code>%tensorflow_version 1.x</code> magic:\n",
              "<a href=\"https://colab.research.google.com/notebooks/tensorflow_version.ipynb\" target=\"_blank\">more info</a>.</p>\n"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TZXj3IsqyzV8",
        "colab_type": "code",
        "outputId": "4d83489b-45bb-40b5-99fa-9cc0f806a674",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "print(tf.__version__)"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "1.15.0\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cp5wfXDx5SPH",
        "colab_type": "text"
      },
      "source": [
        "In addition to the standard libraries we imported above, we'll need to install BERT's python package."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jviywGyWyKsA",
        "colab_type": "code",
        "outputId": "b3b65df7-2821-40df-f62d-1751b8ee24b3",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 119
        }
      },
      "source": [
        "!pip install bert-tensorflow"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting bert-tensorflow\n",
            "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/a6/66/7eb4e8b6ea35b7cc54c322c816f976167a43019750279a8473d355800a93/bert_tensorflow-1.0.1-py2.py3-none-any.whl (67kB)\n",
            "\r\u001b[K     |████▉                           | 10kB 19.5MB/s eta 0:00:01\r\u001b[K     |█████████▊                      | 20kB 3.2MB/s eta 0:00:01\r\u001b[K     |██████████████▋                 | 30kB 4.6MB/s eta 0:00:01\r\u001b[K     |███████████████████▍            | 40kB 3.1MB/s eta 0:00:01\r\u001b[K     |████████████████████████▎       | 51kB 3.7MB/s eta 0:00:01\r\u001b[K     |█████████████████████████████▏  | 61kB 4.4MB/s eta 0:00:01\r\u001b[K     |████████████████████████████████| 71kB 3.6MB/s \n",
            "\u001b[?25hRequirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from bert-tensorflow) (1.12.0)\n",
            "Installing collected packages: bert-tensorflow\n",
            "Successfully installed bert-tensorflow-1.0.1\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hhbGEfwgdEtw",
        "colab_type": "code",
        "outputId": "3547cece-75d7-49ab-a2aa-b46726a5fc29",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        }
      },
      "source": [
        "import bert\n",
        "from bert import run_classifier\n",
        "from bert import optimization\n",
        "from bert import tokenization"
      ],
      "execution_count": 4,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/bert/optimization.py:87: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KVB3eOcjxxm1",
        "colab_type": "text"
      },
      "source": [
        "Below, we'll set an output directory location to store our model output and checkpoints. This can be a local directory, in which case you'd set OUTPUT_DIR to the name of the directory you'd like to create. If you're running this code in Google's hosted Colab, the directory won't persist after the Colab session ends.\n",
        "\n",
        "Alternatively, if you're a GCP user, you can store output in a GCP bucket. To do that, set a directory name in OUTPUT_DIR and the name of the GCP bucket in the BUCKET field.\n",
        "\n",
        "Set DO_DELETE to rewrite the OUTPUT_DIR if it exists. Otherwise, Tensorflow will load existing model checkpoints from that directory (if they exist)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "US_EAnICvP7f",
        "colab_type": "code",
        "outputId": "cad59ace-a1f2-4d31-d4e4-e0f5db85ca91",
        "cellView": "both",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "# Set the output directory for saving model file\n",
        "# Optionally, set a GCP bucket location\n",
        "\n",
        "OUTPUT_DIR = 'Safemail-BERT'#@param {type:\"string\"}\n",
        "#@markdown Whether or not to clear/delete the directory and create a new one\n",
        "DO_DELETE = False #@param {type:\"boolean\"}\n",
        "#@markdown Set USE_BUCKET and BUCKET if you want to (optionally) store model output on GCP bucket.\n",
        "USE_BUCKET = True #@param {type:\"boolean\"}\n",
        "BUCKET = 'poc-bert' #@param {type:\"string\"} \n",
        "DATA_DIR='sm_tp_all_fulltxt/export_data-sm_tp_all_fulltext-2019-10-25T21:21:55.927Z'#@param {type:\"string\"} \n",
        "if USE_BUCKET:\n",
        "  OUTPUT_DIR = 'gs://{}/{}'.format(BUCKET, OUTPUT_DIR)\n",
        "  DATA_DIR= 'gs://{}/{}'.format(BUCKET, DATA_DIR)\n",
        "  from google.colab import auth\n",
        "  auth.authenticate_user()\n",
        "\n",
        "if DO_DELETE:\n",
        "  try:\n",
        "    tf.gfile.DeleteRecursively(OUTPUT_DIR)\n",
        "  except:\n",
        "    # Doesn't matter if the directory didn't exist\n",
        "    pass\n",
        "tf.gfile.MakeDirs(OUTPUT_DIR)\n",
        "print('***** Model output directory: {} *****'.format(OUTPUT_DIR))\n"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "***** Model output directory: gs://poc-bert/Safemail-BERT *****\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rkpihOFcJ-xN",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 629
        },
        "outputId": "7e29030b-96bd-4d8b-efb6-e8c153793816"
      },
      "source": [
        "from io import StringIO\n",
        "def get_lable_csv_file_path():\n",
        "  files=tf.io.gfile.glob('gs://poc-bert/sm_tp_all_fulltxt/export_data-sm_tp_all_fulltext-2019-10-25T21:21:55.927Z/*.csv')\n",
        "  return files\n",
        " \n",
        "def get_df():\n",
        "  data_test=pd.DataFrame()\n",
        "  data_train=pd.DataFrame()\n",
        "  data_validation=pd.DataFrame()\n",
        "  \n",
        "  files=get_lable_csv_file_path()\n",
        "  for file in files:\n",
        "    data_string=tf.compat.v1.gfile.GFile(file).read()\n",
        "    df = pd.read_csv(StringIO(data_string), names=['category', 'file_path','lable'])\n",
        "    #print(f'df.shape:{df.shape}') # gives number of row count\n",
        "    #print(f'columns count: {df.shape[1]}')# gives number of col count\n",
        "    data_test = data_test.append(df[df['category']=='TEST'], sort=False)\n",
        "    data_train = data_train.append(df[df['category']=='TRAIN'], sort=False)\n",
        "    data_validation = data_validation.append(df[df['category']=='VALIDATION'], sort=False)\n",
        "  return data_test,data_train,data_validation\n",
        "\n",
        "def header_data(df,total=5):\n",
        "  cnt=0\n",
        "  for idx , row in df.iterrows():\n",
        "    print(row[\"category\"], row[\"lable\"], row[\"file_path\"])\n",
        "    cnt=cnt+1\n",
        "    if cnt>10:\n",
        "      break\n",
        "\n",
        "data_test,data_train,data_validation=get_df()\n",
        "\n",
        " \n",
        "print(f'data_train total:{data_train.shape[0]}, top 5 :')\n",
        "header_data(data_train)\n",
        "print(f'data_validation total:{data_validation.shape[0]}, top 5 :')\n",
        "header_data(data_validation)\n",
        "print(f'data_test total:{data_test.shape[0]}, top 5 :')\n",
        "header_data(data_test)\n"
      ],
      "execution_count": 100,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "data_train total:20067, top 5 :\n",
            "TRAIN True_Positive gs://poc-bert/sm_tp_all_fulltxt/export_data-sm_tp_all_fulltext-2019-10-25T21:21:55.927Z/files/RNrx4vloqAw.txt\n",
            "TRAIN False_Positives gs://poc-bert/sm_tp_all_fulltxt/export_data-sm_tp_all_fulltext-2019-10-25T21:21:55.927Z/files/UwrVd4Dl9Ac.txt\n",
            "TRAIN True_Positive gs://poc-bert/sm_tp_all_fulltxt/export_data-sm_tp_all_fulltext-2019-10-25T21:21:55.927Z/files/bcPdj7iNWus.txt\n",
            "TRAIN False_Positives gs://poc-bert/sm_tp_all_fulltxt/export_data-sm_tp_all_fulltext-2019-10-25T21:21:55.927Z/files/BCZ0SNqLEmA.txt\n",
            "TRAIN False_Positives gs://poc-bert/sm_tp_all_fulltxt/export_data-sm_tp_all_fulltext-2019-10-25T21:21:55.927Z/files/WI-kpELsmLY.txt\n",
            "TRAIN True_Positive gs://poc-bert/sm_tp_all_fulltxt/export_data-sm_tp_all_fulltext-2019-10-25T21:21:55.927Z/files/LQqnaQaGbP4.txt\n",
            "TRAIN True_Positive gs://poc-bert/sm_tp_all_fulltxt/export_data-sm_tp_all_fulltext-2019-10-25T21:21:55.927Z/files/PrGctfSfy04.txt\n",
            "TRAIN False_Positives gs://poc-bert/sm_tp_all_fulltxt/export_data-sm_tp_all_fulltext-2019-10-25T21:21:55.927Z/files/D5BoP8nod3w.txt\n",
            "TRAIN True_Positive gs://poc-bert/sm_tp_all_fulltxt/export_data-sm_tp_all_fulltext-2019-10-25T21:21:55.927Z/files/Rdd-soXPQfU.txt\n",
            "TRAIN True_Positive gs://poc-bert/sm_tp_all_fulltxt/export_data-sm_tp_all_fulltext-2019-10-25T21:21:55.927Z/files/OfTkBEt9Xuo.txt\n",
            "TRAIN True_Positive gs://poc-bert/sm_tp_all_fulltxt/export_data-sm_tp_all_fulltext-2019-10-25T21:21:55.927Z/files/RWj1kt9Kdy0.txt\n",
            "data_validation total:2518, top 5 :\n",
            "VALIDATION False_Positives gs://poc-bert/sm_tp_all_fulltxt/export_data-sm_tp_all_fulltext-2019-10-25T21:21:55.927Z/files/dHg7Btu_08A.txt\n",
            "VALIDATION False_Positives gs://poc-bert/sm_tp_all_fulltxt/export_data-sm_tp_all_fulltext-2019-10-25T21:21:55.927Z/files/M5btcQ_mzxg.txt\n",
            "VALIDATION True_Positive gs://poc-bert/sm_tp_all_fulltxt/export_data-sm_tp_all_fulltext-2019-10-25T21:21:55.927Z/files/KgtVlSOfovE.txt\n",
            "VALIDATION False_Positives gs://poc-bert/sm_tp_all_fulltxt/export_data-sm_tp_all_fulltext-2019-10-25T21:21:55.927Z/files/KfSbvzt68Xw.txt\n",
            "VALIDATION False_Positives gs://poc-bert/sm_tp_all_fulltxt/export_data-sm_tp_all_fulltext-2019-10-25T21:21:55.927Z/files/S7w6bjXyFRU.txt\n",
            "VALIDATION True_Positive gs://poc-bert/sm_tp_all_fulltxt/export_data-sm_tp_all_fulltext-2019-10-25T21:21:55.927Z/files/XFE9VOWJ4jk.txt\n",
            "VALIDATION True_Positive gs://poc-bert/sm_tp_all_fulltxt/export_data-sm_tp_all_fulltext-2019-10-25T21:21:55.927Z/files/Gm_ysELfF0U.txt\n",
            "VALIDATION True_Positive gs://poc-bert/sm_tp_all_fulltxt/export_data-sm_tp_all_fulltext-2019-10-25T21:21:55.927Z/files/ZrYhYvTesL8.txt\n",
            "VALIDATION True_Positive gs://poc-bert/sm_tp_all_fulltxt/export_data-sm_tp_all_fulltext-2019-10-25T21:21:55.927Z/files/VQLb_IWoAkM.txt\n",
            "VALIDATION False_Positives gs://poc-bert/sm_tp_all_fulltxt/export_data-sm_tp_all_fulltext-2019-10-25T21:21:55.927Z/files/PrzS94-bDcQ.txt\n",
            "VALIDATION False_Positives gs://poc-bert/sm_tp_all_fulltxt/export_data-sm_tp_all_fulltext-2019-10-25T21:21:55.927Z/files/dSTpqIYjjT0.txt\n",
            "data_test total:2560, top 5 :\n",
            "TEST True_Positive gs://poc-bert/sm_tp_all_fulltxt/export_data-sm_tp_all_fulltext-2019-10-25T21:21:55.927Z/files/IddU5r6_Q9U.txt\n",
            "TEST True_Positive gs://poc-bert/sm_tp_all_fulltxt/export_data-sm_tp_all_fulltext-2019-10-25T21:21:55.927Z/files/GgavsEsTBDs.txt\n",
            "TEST False_Positives gs://poc-bert/sm_tp_all_fulltxt/export_data-sm_tp_all_fulltext-2019-10-25T21:21:55.927Z/files/EXAudR4KO9c.txt\n",
            "TEST True_Positive gs://poc-bert/sm_tp_all_fulltxt/export_data-sm_tp_all_fulltext-2019-10-25T21:21:55.927Z/files/Xryw_hHP0Ck.txt\n",
            "TEST True_Positive gs://poc-bert/sm_tp_all_fulltxt/export_data-sm_tp_all_fulltext-2019-10-25T21:21:55.927Z/files/SX_lNrLnVKQ.txt\n",
            "TEST False_Positives gs://poc-bert/sm_tp_all_fulltxt/export_data-sm_tp_all_fulltext-2019-10-25T21:21:55.927Z/files/ZfviJWFH-_U.txt\n",
            "TEST True_Positive gs://poc-bert/sm_tp_all_fulltxt/export_data-sm_tp_all_fulltext-2019-10-25T21:21:55.927Z/files/MI80jvOINd8.txt\n",
            "TEST False_Positives gs://poc-bert/sm_tp_all_fulltxt/export_data-sm_tp_all_fulltext-2019-10-25T21:21:55.927Z/files/fZZdSJCQW5A.txt\n",
            "TEST False_Positives gs://poc-bert/sm_tp_all_fulltxt/export_data-sm_tp_all_fulltext-2019-10-25T21:21:55.927Z/files/T7Nt3r7v-8c.txt\n",
            "TEST True_Positive gs://poc-bert/sm_tp_all_fulltxt/export_data-sm_tp_all_fulltext-2019-10-25T21:21:55.927Z/files/V3HO9q-o--o.txt\n",
            "TEST False_Positives gs://poc-bert/sm_tp_all_fulltxt/export_data-sm_tp_all_fulltext-2019-10-25T21:21:55.927Z/files/czirH5Mlinc.txt\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wHOzPU_F8YAH",
        "colab_type": "text"
      },
      "source": [
        "get data from bucket"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0mDf0sVE5S0M",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "3cbe58c1-054b-4033-ed02-c166f95d0097"
      },
      "source": [
        "def get_lable_file():\n",
        "  items=tf.compat.v1.io.gfile.listdir(DATA_DIR)\n",
        "  csv_files=[f for f in items if f.endswith('.csv') ]\n",
        "  return files\n",
        "files=get_lable_file() \n",
        "print(files)"
      ],
      "execution_count": 80,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "['text_classification_1.csv', 'text_classification_2.csv', 'text_classification_3.csv']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pmFYvkylMwXn",
        "colab_type": "text"
      },
      "source": [
        "#Data"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MC_w8SRqN0fr",
        "colab_type": "text"
      },
      "source": [
        "First, let's download the dataset, hosted by Stanford. The code below, which downloads, extracts, and imports the IMDB Large Movie Review Dataset, is borrowed from [this Tensorflow tutorial](https://www.tensorflow.org/hub/tutorials/text_classification_with_tf_hub)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fom_ff20gyy6",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from tensorflow import keras\n",
        "import os\n",
        "import re\n",
        "\n",
        "# Load all files from a directory in a DataFrame.\n",
        "def load_directory_data(directory):\n",
        "  data = {}\n",
        "  data[\"sentence\"] = []\n",
        "  data[\"sentiment\"] = []\n",
        "  data[\"file_path\"] = []\n",
        "  for file_path in os.listdir(directory):\n",
        "    with tf.gfile.GFile(os.path.join(directory, file_path), \"r\") as f:\n",
        "      data[\"sentence\"].append(f.read())\n",
        "      data[\"sentiment\"].append(re.match(\"\\d+_(\\d+)\\.txt\", file_path).group(1))\n",
        "      data[\"file_path\"].append(file_path)\n",
        "      \n",
        "  return pd.DataFrame.from_dict(data)\n",
        "\n",
        "# Merge positive and negative examples, add a polarity column and shuffle.\n",
        "def load_dataset(directory):\n",
        "  pos_df = load_directory_data(os.path.join(directory, \"pos\"))\n",
        "  neg_df = load_directory_data(os.path.join(directory, \"neg\"))\n",
        "  pos_df[\"polarity\"] = 1\n",
        "  neg_df[\"polarity\"] = 0\n",
        "  return pd.concat([pos_df, neg_df]).sample(frac=1).reset_index(drop=True)\n",
        "\n",
        "# Download and process the dataset files.\n",
        "def download_and_load_datasets(force_download=False):\n",
        "  dataset = tf.keras.utils.get_file(\n",
        "      fname=\"aclImdb.tar.gz\", \n",
        "      origin=\"http://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz\", \n",
        "      extract=True)\n",
        "  \n",
        "  train_df = load_dataset(os.path.join(os.path.dirname(dataset), \n",
        "                                       \"aclImdb\", \"train\"))\n",
        "  test_df = load_dataset(os.path.join(os.path.dirname(dataset), \n",
        "                                      \"aclImdb\", \"test\"))\n",
        "  \n",
        "  return train_df, test_df\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2abfwdn-g135",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "train, test = download_and_load_datasets()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9f4KnNVvwt7r",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "tt=train['sentence']\n",
        "ss=train['sentiment']\n",
        "pp=train['polarity']\n",
        "fp=train['file_path']\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Hd44CdYkwy8Z",
        "colab_type": "code",
        "outputId": "485f3b58-a98c-4b3c-dc37-21dae2b43cc2",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 105
        }
      },
      "source": [
        "idx=100\n",
        "print(tt[idx])\n",
        "print(ss[idx])\n",
        "print(pp[idx])\n",
        "print(fp[idx])\n",
        " "
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Around the late 1970's, animator Don Bluth, frustrated with the output his company, Disney was churning, defected from the Mouse House to form his own studio. His first production, THE SECRET OF NIMH, was a brilliant feature that still holds up well to this day. This was followed by AN American TAIL and THE LAND BEFORE TIME, both of which were made under the involvement of Steven Spielberg and were commercially successful. Although none of those two films had the dark adult appeal of NIMH, they still are very charming, enjoyable features for both children and grown-ups. But before long, Don Bluth had his first major misfire with ALL DOGS GO TO HEAVEN; critics were especially harsh on this film, and matters weren't helped by the fact that it opened alongside Disney's THE LITTLE MERMAID.<br /><br />Considering that the movie has such a friendly-sounding title, one would expect ALL DOGS GO TO HEAVEN to be pleasant family fare. Instead Bluth provides a surprisingly dark story involving gambling, deceit, crime, mistreatment, and murder. That itself is not a problem for an animated feature per say, but it does call into question over whether the film is for children. On the other hand, it's hard to say whether adults will find much to enjoy in ALL DOGS GO TO HEAVEN. In short, it's a movie with a major identity crisis.<br /><br />Set in a dreary junkyard of New Orleans, the movie starts out when Charlie B. Barkin, a rough-and-tumble German shepherd, is run over by a car courtesy of his former gambling casino partner, a nasty, cigar-puffing pitbull, Carface. Before you know it, Charlie finds himself in heaven, albeit by default. Here a whippet angel, Annabelle, tells him that \"all dogs go to heaven because unlike people, dogs are usually loyal and kind.\" This line represents the confused nature of the movie, since the dogs in the movie, the whippet aside, are presented as anything but.<br /><br />Upon realizing that he's been murdered, Charlie steals his way back to Earth and plots to get even with Carface. With the reluctant help of his dachshund pal Itchy, Charlie \"rescues\" Carface's prize, AnneMarie, a human girl who can talk to animals (in order to predict who will win the rat races). Charlie claims that he will help the little cutie find her a family, but in reality he is using her skills to win fortunes at the race so that he can build a more elaborate casino of his own to bring Carface down. Although he refuses to admit it, Charlie does grow to love AnneMarie...<br /><br />The concept of the story isn't as problematic as the execution. Aside from the human girl AnneMarie and a flamboyant musical alligator who appears about three-quarters through (with the vocal pipes of Ken Page), none of the other characters emerge as likable, nor frankly, are even worth caring about. Unfortunately, that also applies to Charlie; in trying to make him an anti-hero, the script (composed by more than ten writers) only succeeds in rendering the character TOO unlovable. As such, the audience feels no empathy for Charlie, and worse, his redemption at the end of the movie does not come across as convincing. (Further damaging to the character is the disappointingly uncharismatic vocal performance from Burt Reynolds.) Besides the lack of an endearing lead, the movie's other problem is in the structure of the story. The slowly-paced plot jumps all over the place and makes a habit of throwing in extra scenes which serve no purpose but to pad out the movie's running time. The aforementioned musical alligator (who resides in a danky sewer infested with native rats) seems to have been thrown in from nowhere, as does a scene where Charlie tries to show his generosity to AnneMarie by feeding a pack of pastel-colored pups pizza. The whole screenplay feels like a rough first draft; a bit more polish could have made this a tighter, impactful story.<br /><br />Matters are not helped by the lackluster musical numbers by Charlie Strouse and T.J. Kuenster (AnneMarie's song and the gator's ballad are the only good ones; the latter in particular benefits from Ken Page's mellifluous vocal) or the uneven voice cast. As mentioned, Burt Reynolds' stiff and lifeless Charlie detracts from his already unlikeable character even further (the only exception is a fiery confession to Itchy about his true intentions toward the end). Dom DeLuise as Itchy is pretty good, but he's had better roles, notably Tiger in AN American TAIL and Jeremy in THE SECRET OF NIMH. Ken Page, as mentioned, is awesome in anything he does, but his character has such a small part that his overall contribution is unremarkable at best. Similarly wasted are Loni Anderson (as a collie who once sired a litter with Charlie), Melba Moore, and Charles Nelson Reilly. Judith Barsi as AnneMarie is probably the only voice that comes across as truly memorable, partially because her character is the sole legitimately likable one in this depressing and joyless show.<br /><br />Barsi aside, the only real positive about ALL DOGS GO TO HEAVEN is the animation. Technically, this film has some of the most imaginative visuals from Bluth's team (by 1980's standards, that is), particularly a frightening scene where Charlie has a nightmare about ending up in a fiery underworld ruled by a gargantuan satanic canine-demon. If anything, the movie is more of a triumph of animation than storytelling.<br /><br />On the whole, however, I cannot recommend ALL DOGS GO TO HEAVEN as good entertainment. Even though I recognize that the movie has its fans and the climax does admittingly provide some energy and a moving conclusion, the overall package is not in the same league as Bluth's better efforts. Animation buffs will marvel at the lush artistry, but by the time it's over, ALL DOGS GO TO HEAVEN could very well leave a bad taste in your mouth.\n",
            "4\n",
            "0\n",
            "539_4.txt\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XA8WHJgzhIZf",
        "colab_type": "text"
      },
      "source": [
        "To keep training fast, we'll take a sample of 5000 train and test examples, respectively."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lw_F488eixTV",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "train = train.sample(5000)\n",
        "test = test.sample(5000)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "prRQM8pDi8xI",
        "colab_type": "code",
        "outputId": "873fd1ae-22d6-404f-835d-9b5e284df6af",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "train.columns"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Index(['sentence', 'sentiment', 'file_path', 'polarity'], dtype='object')"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 37
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sfRnHSz3iSXz",
        "colab_type": "text"
      },
      "source": [
        "For us, our input data is the 'sentence' column and our label is the 'polarity' column (0, 1 for negative and positive, respecitvely)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IuMOGwFui4it",
        "colab_type": "code",
        "outputId": "cd1f5663-124a-47cb-e963-a2c9cae56585",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        }
      },
      "source": [
        "DATA_COLUMN = 'sentence'\n",
        "LABEL_COLUMN = 'polarity'\n",
        "# label_list is the list of labels, i.e. True, False or 0, 1 or 'dog', 'cat'\n",
        "label_list = [0, 1]\n",
        "print(len(train))\n",
        "print(type(train))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "5000\n",
            "<class 'pandas.core.frame.DataFrame'>\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "V399W0rqNJ-Z",
        "colab_type": "text"
      },
      "source": [
        "#Data Preprocessing\n",
        "We'll need to transform our data into a format BERT understands. This involves two steps. First, we create  `InputExample`'s using the constructor provided in the BERT library.\n",
        "\n",
        "- `text_a` is the text we want to classify, which in this case, is the `Request` field in our Dataframe. \n",
        "- `text_b` is used if we're training a model to understand the relationship between sentences (i.e. is `text_b` a translation of `text_a`? Is `text_b` an answer to the question asked by `text_a`?). This doesn't apply to our task, so we can leave `text_b` blank.\n",
        "- `label` is the label for our example, i.e. True, False"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "p9gEt5SmM6i6",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Use the InputExample class from BERT's run_classifier code to create examples from the data\n",
        "train_InputExamples = train.apply(lambda x: bert.run_classifier.InputExample(guid=None, # Globally unique ID for bookkeeping, unused in this example\n",
        "                                                                   text_a = x[DATA_COLUMN], \n",
        "                                                                   text_b = None, \n",
        "                                                                   label = x[LABEL_COLUMN]), axis = 1)\n",
        "\n",
        "test_InputExamples = test.apply(lambda x: bert.run_classifier.InputExample(guid=None, \n",
        "                                                                   text_a = x[DATA_COLUMN], \n",
        "                                                                   text_b = None, \n",
        "                                                                   label = x[LABEL_COLUMN]), axis = 1)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SCZWZtKxObjh",
        "colab_type": "text"
      },
      "source": [
        "Next, we need to preprocess our data so that it matches the data BERT was trained on. For this, we'll need to do a couple of things (but don't worry--this is also included in the Python library):\n",
        "\n",
        "\n",
        "1. Lowercase our text (if we're using a BERT lowercase model)\n",
        "2. Tokenize it (i.e. \"sally says hi\" -> [\"sally\", \"says\", \"hi\"])\n",
        "3. Break words into WordPieces (i.e. \"calling\" -> [\"call\", \"##ing\"])\n",
        "4. Map our words to indexes using a vocab file that BERT provides\n",
        "5. Add special \"CLS\" and \"SEP\" tokens (see the [readme](https://github.com/google-research/bert))\n",
        "6. Append \"index\" and \"segment\" tokens to each input (see the [BERT paper](https://arxiv.org/pdf/1810.04805.pdf))\n",
        "\n",
        "Happily, we don't have to worry about most of these details.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qMWiDtpyQSoU",
        "colab_type": "text"
      },
      "source": [
        "To start, we'll need to load a vocabulary file and lowercasing information directly from the BERT tf hub module:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IhJSe0QHNG7U",
        "colab_type": "code",
        "outputId": "c372d28e-e459-4f45-9df8-096de60d81be",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 85
        }
      },
      "source": [
        "# This is a path to an uncased (all lowercase) version of BERT\n",
        "BERT_MODEL_HUB = \"https://tfhub.dev/google/bert_uncased_L-12_H-768_A-12/1\"\n",
        "\n",
        "def create_tokenizer_from_hub_module():\n",
        "  \"\"\"Get the vocab file and casing info from the Hub module.\"\"\"\n",
        "  with tf.Graph().as_default():\n",
        "    bert_module = hub.Module(BERT_MODEL_HUB)\n",
        "    tokenization_info = bert_module(signature=\"tokenization_info\", as_dict=True)\n",
        "    import pprint\n",
        "    pprint.pprint(tokenization_info[\"do_lower_case\"])\n",
        "    pprint.pprint(tokenization_info[\"vocab_file\"])\n",
        "    with tf.Session() as sess:\n",
        "      vocab_file, do_lower_case = sess.run([tokenization_info[\"vocab_file\"],\n",
        "                                            tokenization_info[\"do_lower_case\"]])\n",
        "      \n",
        "  return bert.tokenization.FullTokenizer(\n",
        "      vocab_file=vocab_file, do_lower_case=do_lower_case)\n",
        "\n",
        "tokenizer = create_tokenizer_from_hub_module()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Saver not created because there are no variables in the graph to restore\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Saver not created because there are no variables in the graph to restore\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "<tf.Tensor 'module_apply_tokenization_info/Const:0' shape=() dtype=bool>\n",
            "<tf.Tensor 'module_apply_tokenization_info/vocab_file:0' shape=() dtype=string>\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "z4oFkhpZBDKm",
        "colab_type": "text"
      },
      "source": [
        "Great--we just learned that the BERT model we're using expects lowercase data (that's what stored in tokenization_info[\"do_lower_case\"]) and we also loaded BERT's vocab file. We also created a tokenizer, which breaks words into word pieces:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dsBo6RCtQmwx",
        "colab_type": "code",
        "outputId": "c9640759-ab40-49b5-ab5b-f6c92a091032",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 221
        }
      },
      "source": [
        "tokenizer.tokenize(\"This here's an example of using the BERT tokenizer\")"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['this',\n",
              " 'here',\n",
              " \"'\",\n",
              " 's',\n",
              " 'an',\n",
              " 'example',\n",
              " 'of',\n",
              " 'using',\n",
              " 'the',\n",
              " 'bert',\n",
              " 'token',\n",
              " '##izer']"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 45
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0OEzfFIt6GIc",
        "colab_type": "text"
      },
      "source": [
        "Using our tokenizer, we'll call `run_classifier.convert_examples_to_features` on our InputExamples to convert them into features BERT understands."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LL5W8gEGRTAf",
        "colab_type": "code",
        "outputId": "2e9e8681-f6f2-49d2-a603-7521b7d475a2",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "# We'll set sequences to be at most 128 tokens long.\n",
        "MAX_SEQ_LENGTH = 128\n",
        "# Convert our train and test features to InputFeatures that BERT understands.\n",
        "train_features = bert.run_classifier.convert_examples_to_features(train_InputExamples, label_list, MAX_SEQ_LENGTH, tokenizer)\n",
        "test_features = bert.run_classifier.convert_examples_to_features(test_InputExamples, label_list, MAX_SEQ_LENGTH, tokenizer)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/bert/run_classifier.py:774: The name tf.logging.info is deprecated. Please use tf.compat.v1.logging.info instead.\n",
            "\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/bert/run_classifier.py:774: The name tf.logging.info is deprecated. Please use tf.compat.v1.logging.info instead.\n",
            "\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Writing example 0 of 5000\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Writing example 0 of 5000\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Example ***\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Example ***\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:guid: None\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:guid: None\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:tokens: [CLS] going into this movie , i had heard good things about it . coming out of it , i wasn ' t really amazed nor disappointed . simon peg ##g plays a rather childish character much like his other movies . there were a couple of laughs here and there - - nothing too funny . probably my favorite parts of the movie is when he dances in the club scene . i totally gotta try that out next time i find myself in a club . a couple of stars here and there including : megan fox , ki ##rsten dun ##st , that chick from x - files , and jeff bridges . i found it quite amusing to see a cameo appearance of [SEP]\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:tokens: [CLS] going into this movie , i had heard good things about it . coming out of it , i wasn ' t really amazed nor disappointed . simon peg ##g plays a rather childish character much like his other movies . there were a couple of laughs here and there - - nothing too funny . probably my favorite parts of the movie is when he dances in the club scene . i totally gotta try that out next time i find myself in a club . a couple of stars here and there including : megan fox , ki ##rsten dun ##st , that chick from x - files , and jeff bridges . i found it quite amusing to see a cameo appearance of [SEP]\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_ids: 101 2183 2046 2023 3185 1010 1045 2018 2657 2204 2477 2055 2009 1012 2746 2041 1997 2009 1010 1045 2347 1005 1056 2428 15261 4496 9364 1012 4079 25039 2290 3248 1037 2738 24282 2839 2172 2066 2010 2060 5691 1012 2045 2020 1037 3232 1997 11680 2182 1998 2045 1011 1011 2498 2205 6057 1012 2763 2026 5440 3033 1997 1996 3185 2003 2043 2002 11278 1999 1996 2252 3496 1012 1045 6135 10657 3046 2008 2041 2279 2051 1045 2424 2870 1999 1037 2252 1012 1037 3232 1997 3340 2182 1998 2045 2164 1024 12756 4419 1010 11382 19020 24654 3367 1010 2008 14556 2013 1060 1011 6764 1010 1998 5076 7346 1012 1045 2179 2009 3243 19142 2000 2156 1037 12081 3311 1997 102\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_ids: 101 2183 2046 2023 3185 1010 1045 2018 2657 2204 2477 2055 2009 1012 2746 2041 1997 2009 1010 1045 2347 1005 1056 2428 15261 4496 9364 1012 4079 25039 2290 3248 1037 2738 24282 2839 2172 2066 2010 2060 5691 1012 2045 2020 1037 3232 1997 11680 2182 1998 2045 1011 1011 2498 2205 6057 1012 2763 2026 5440 3033 1997 1996 3185 2003 2043 2002 11278 1999 1996 2252 3496 1012 1045 6135 10657 3046 2008 2041 2279 2051 1045 2424 2870 1999 1037 2252 1012 1037 3232 1997 3340 2182 1998 2045 2164 1024 12756 4419 1010 11382 19020 24654 3367 1010 2008 14556 2013 1060 1011 6764 1010 1998 5076 7346 1012 1045 2179 2009 3243 19142 2000 2156 1037 12081 3311 1997 102\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:label: 1 (id = 1)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:label: 1 (id = 1)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Example ***\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Example ***\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:guid: None\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:guid: None\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:tokens: [CLS] the story : on the island tex ##el , photographer bob , who makes a photo shoot for a magazine , meets the mysterious kathleen . her free spirit and lust for life int ##rigues bob , who has suffered a very traumatic experience shortly before . her life is not so simple as it seems , however . through kathleen , bob gets en ##tangled in a dangerous network . will kathleen be able to win his trust ? < br / > < br / > review : the dialogue in this movie is very natural and the story un ##fold ##s nicely although it stays a bit on the surface and it would have been nice if the character ' s ' psychology [SEP]\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:tokens: [CLS] the story : on the island tex ##el , photographer bob , who makes a photo shoot for a magazine , meets the mysterious kathleen . her free spirit and lust for life int ##rigues bob , who has suffered a very traumatic experience shortly before . her life is not so simple as it seems , however . through kathleen , bob gets en ##tangled in a dangerous network . will kathleen be able to win his trust ? < br / > < br / > review : the dialogue in this movie is very natural and the story un ##fold ##s nicely although it stays a bit on the surface and it would have been nice if the character ' s ' psychology [SEP]\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_ids: 101 1996 2466 1024 2006 1996 2479 16060 2884 1010 8088 3960 1010 2040 3084 1037 6302 5607 2005 1037 2932 1010 6010 1996 8075 14559 1012 2014 2489 4382 1998 11516 2005 2166 20014 22934 3960 1010 2040 2038 4265 1037 2200 19686 3325 3859 2077 1012 2014 2166 2003 2025 2061 3722 2004 2009 3849 1010 2174 1012 2083 14559 1010 3960 4152 4372 27898 1999 1037 4795 2897 1012 2097 14559 2022 2583 2000 2663 2010 3404 1029 1026 7987 1013 1028 1026 7987 1013 1028 3319 1024 1996 7982 1999 2023 3185 2003 2200 3019 1998 1996 2466 4895 10371 2015 19957 2348 2009 12237 1037 2978 2006 1996 3302 1998 2009 2052 2031 2042 3835 2065 1996 2839 1005 1055 1005 6825 102\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_ids: 101 1996 2466 1024 2006 1996 2479 16060 2884 1010 8088 3960 1010 2040 3084 1037 6302 5607 2005 1037 2932 1010 6010 1996 8075 14559 1012 2014 2489 4382 1998 11516 2005 2166 20014 22934 3960 1010 2040 2038 4265 1037 2200 19686 3325 3859 2077 1012 2014 2166 2003 2025 2061 3722 2004 2009 3849 1010 2174 1012 2083 14559 1010 3960 4152 4372 27898 1999 1037 4795 2897 1012 2097 14559 2022 2583 2000 2663 2010 3404 1029 1026 7987 1013 1028 1026 7987 1013 1028 3319 1024 1996 7982 1999 2023 3185 2003 2200 3019 1998 1996 2466 4895 10371 2015 19957 2348 2009 12237 1037 2978 2006 1996 3302 1998 2009 2052 2031 2042 3835 2065 1996 2839 1005 1055 1005 6825 102\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:label: 1 (id = 1)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:label: 1 (id = 1)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Example ***\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Example ***\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:guid: None\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:guid: None\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:tokens: [CLS] mu ##lho ##lland drive made me the definitive fan of david lynch . he ' s a modern genius , because he ' s not only a film - maker . his stories and his style have a spell that cross the screen . so the straight story was quite a surprise to me , with its easy to follow storyline and sunny sets . still , lynch is there , and , while this is far from his best , it ' s a film not to be missed . late richard far ##ns ##worth ' s performance is one of the reasons . < br / > < br / > 8 / 10 [SEP]\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:tokens: [CLS] mu ##lho ##lland drive made me the definitive fan of david lynch . he ' s a modern genius , because he ' s not only a film - maker . his stories and his style have a spell that cross the screen . so the straight story was quite a surprise to me , with its easy to follow storyline and sunny sets . still , lynch is there , and , while this is far from his best , it ' s a film not to be missed . late richard far ##ns ##worth ' s performance is one of the reasons . < br / > < br / > 8 / 10 [SEP]\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_ids: 101 14163 28061 26087 3298 2081 2033 1996 15764 5470 1997 2585 11404 1012 2002 1005 1055 1037 2715 11067 1010 2138 2002 1005 1055 2025 2069 1037 2143 1011 9338 1012 2010 3441 1998 2010 2806 2031 1037 6297 2008 2892 1996 3898 1012 2061 1996 3442 2466 2001 3243 1037 4474 2000 2033 1010 2007 2049 3733 2000 3582 9994 1998 11559 4520 1012 2145 1010 11404 2003 2045 1010 1998 1010 2096 2023 2003 2521 2013 2010 2190 1010 2009 1005 1055 1037 2143 2025 2000 2022 4771 1012 2397 2957 2521 3619 5172 1005 1055 2836 2003 2028 1997 1996 4436 1012 1026 7987 1013 1028 1026 7987 1013 1028 1022 1013 2184 102 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_ids: 101 14163 28061 26087 3298 2081 2033 1996 15764 5470 1997 2585 11404 1012 2002 1005 1055 1037 2715 11067 1010 2138 2002 1005 1055 2025 2069 1037 2143 1011 9338 1012 2010 3441 1998 2010 2806 2031 1037 6297 2008 2892 1996 3898 1012 2061 1996 3442 2466 2001 3243 1037 4474 2000 2033 1010 2007 2049 3733 2000 3582 9994 1998 11559 4520 1012 2145 1010 11404 2003 2045 1010 1998 1010 2096 2023 2003 2521 2013 2010 2190 1010 2009 1005 1055 1037 2143 2025 2000 2022 4771 1012 2397 2957 2521 3619 5172 1005 1055 2836 2003 2028 1997 1996 4436 1012 1026 7987 1013 1028 1026 7987 1013 1028 1022 1013 2184 102 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:label: 1 (id = 1)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:label: 1 (id = 1)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Example ***\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Example ***\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:guid: None\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:guid: None\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:tokens: [CLS] okay i saw the sneak preview of this stupid movie . first off the movie is so posed and not real , they are all acting . they can ' t sing . they are way too full of themselves . its awful . yes kids like 8 to 10 might enjoy but its really stupid . i mean they say their manager is a kid . and there record label is fake . its stupid . don ' t see it . < br / > < br / > as for the set up and directing , not so bad . it is a cute documentary but it documents a stupid thing . < br / > < br / > only see this [SEP]\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:tokens: [CLS] okay i saw the sneak preview of this stupid movie . first off the movie is so posed and not real , they are all acting . they can ' t sing . they are way too full of themselves . its awful . yes kids like 8 to 10 might enjoy but its really stupid . i mean they say their manager is a kid . and there record label is fake . its stupid . don ' t see it . < br / > < br / > as for the set up and directing , not so bad . it is a cute documentary but it documents a stupid thing . < br / > < br / > only see this [SEP]\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_ids: 101 3100 1045 2387 1996 13583 19236 1997 2023 5236 3185 1012 2034 2125 1996 3185 2003 2061 13686 1998 2025 2613 1010 2027 2024 2035 3772 1012 2027 2064 1005 1056 6170 1012 2027 2024 2126 2205 2440 1997 3209 1012 2049 9643 1012 2748 4268 2066 1022 2000 2184 2453 5959 2021 2049 2428 5236 1012 1045 2812 2027 2360 2037 3208 2003 1037 4845 1012 1998 2045 2501 3830 2003 8275 1012 2049 5236 1012 2123 1005 1056 2156 2009 1012 1026 7987 1013 1028 1026 7987 1013 1028 2004 2005 1996 2275 2039 1998 9855 1010 2025 2061 2919 1012 2009 2003 1037 10140 4516 2021 2009 5491 1037 5236 2518 1012 1026 7987 1013 1028 1026 7987 1013 1028 2069 2156 2023 102\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_ids: 101 3100 1045 2387 1996 13583 19236 1997 2023 5236 3185 1012 2034 2125 1996 3185 2003 2061 13686 1998 2025 2613 1010 2027 2024 2035 3772 1012 2027 2064 1005 1056 6170 1012 2027 2024 2126 2205 2440 1997 3209 1012 2049 9643 1012 2748 4268 2066 1022 2000 2184 2453 5959 2021 2049 2428 5236 1012 1045 2812 2027 2360 2037 3208 2003 1037 4845 1012 1998 2045 2501 3830 2003 8275 1012 2049 5236 1012 2123 1005 1056 2156 2009 1012 1026 7987 1013 1028 1026 7987 1013 1028 2004 2005 1996 2275 2039 1998 9855 1010 2025 2061 2919 1012 2009 2003 1037 10140 4516 2021 2009 5491 1037 5236 2518 1012 1026 7987 1013 1028 1026 7987 1013 1028 2069 2156 2023 102\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:label: 0 (id = 0)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:label: 0 (id = 0)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Example ***\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Example ***\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:guid: None\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:guid: None\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:tokens: [CLS] i love watching jerry as much as the rest of the world , but this poor excuse for a soft - core porn ##o flick is needles ##sl ##y offensive , lacks anything resembling wit , and serves merely as a vehicle of self - promotion for springer . even though it runs a quick 90 minutes , the film drag ##s hideous ##ly , and i should have had the common sense to walk out . simply at ##ro ##cious . [SEP]\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:tokens: [CLS] i love watching jerry as much as the rest of the world , but this poor excuse for a soft - core porn ##o flick is needles ##sl ##y offensive , lacks anything resembling wit , and serves merely as a vehicle of self - promotion for springer . even though it runs a quick 90 minutes , the film drag ##s hideous ##ly , and i should have had the common sense to walk out . simply at ##ro ##cious . [SEP]\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_ids: 101 1045 2293 3666 6128 2004 2172 2004 1996 2717 1997 1996 2088 1010 2021 2023 3532 8016 2005 1037 3730 1011 4563 22555 2080 17312 2003 17044 14540 2100 5805 1010 14087 2505 15525 15966 1010 1998 4240 6414 2004 1037 4316 1997 2969 1011 4712 2005 17481 1012 2130 2295 2009 3216 1037 4248 3938 2781 1010 1996 2143 8011 2015 22293 2135 1010 1998 1045 2323 2031 2018 1996 2691 3168 2000 3328 2041 1012 3432 2012 3217 18436 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_ids: 101 1045 2293 3666 6128 2004 2172 2004 1996 2717 1997 1996 2088 1010 2021 2023 3532 8016 2005 1037 3730 1011 4563 22555 2080 17312 2003 17044 14540 2100 5805 1010 14087 2505 15525 15966 1010 1998 4240 6414 2004 1037 4316 1997 2969 1011 4712 2005 17481 1012 2130 2295 2009 3216 1037 4248 3938 2781 1010 1996 2143 8011 2015 22293 2135 1010 1998 1045 2323 2031 2018 1996 2691 3168 2000 3328 2041 1012 3432 2012 3217 18436 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:label: 0 (id = 0)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:label: 0 (id = 0)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Writing example 0 of 5000\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Writing example 0 of 5000\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Example ***\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Example ***\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:guid: None\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:guid: None\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:tokens: [CLS] this is the movie that finally pushed me over the line into registering with im ##db so that i could vote for ( and comment on ) it . i ' ve only recently come to appreciate well - produced \" war \" movies , and this is one of the most thoughtful i ' ve seen . < br / > < br / > \" stunning \" is the word that comes to mind when i think of this viewing experience . my husband and i watched this film last night for the first time . it is gently moving , yet exciting at the same time ( not a contradiction ) . this story in the hands of hollywood could have become just [SEP]\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:tokens: [CLS] this is the movie that finally pushed me over the line into registering with im ##db so that i could vote for ( and comment on ) it . i ' ve only recently come to appreciate well - produced \" war \" movies , and this is one of the most thoughtful i ' ve seen . < br / > < br / > \" stunning \" is the word that comes to mind when i think of this viewing experience . my husband and i watched this film last night for the first time . it is gently moving , yet exciting at the same time ( not a contradiction ) . this story in the hands of hollywood could have become just [SEP]\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_ids: 101 2023 2003 1996 3185 2008 2633 3724 2033 2058 1996 2240 2046 25719 2007 10047 18939 2061 2008 1045 2071 3789 2005 1006 1998 7615 2006 1007 2009 1012 1045 1005 2310 2069 3728 2272 2000 9120 2092 1011 2550 1000 2162 1000 5691 1010 1998 2023 2003 2028 1997 1996 2087 16465 1045 1005 2310 2464 1012 1026 7987 1013 1028 1026 7987 1013 1028 1000 14726 1000 2003 1996 2773 2008 3310 2000 2568 2043 1045 2228 1997 2023 10523 3325 1012 2026 3129 1998 1045 3427 2023 2143 2197 2305 2005 1996 2034 2051 1012 2009 2003 5251 3048 1010 2664 10990 2012 1996 2168 2051 1006 2025 1037 26917 1007 1012 2023 2466 1999 1996 2398 1997 5365 2071 2031 2468 2074 102\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_ids: 101 2023 2003 1996 3185 2008 2633 3724 2033 2058 1996 2240 2046 25719 2007 10047 18939 2061 2008 1045 2071 3789 2005 1006 1998 7615 2006 1007 2009 1012 1045 1005 2310 2069 3728 2272 2000 9120 2092 1011 2550 1000 2162 1000 5691 1010 1998 2023 2003 2028 1997 1996 2087 16465 1045 1005 2310 2464 1012 1026 7987 1013 1028 1026 7987 1013 1028 1000 14726 1000 2003 1996 2773 2008 3310 2000 2568 2043 1045 2228 1997 2023 10523 3325 1012 2026 3129 1998 1045 3427 2023 2143 2197 2305 2005 1996 2034 2051 1012 2009 2003 5251 3048 1010 2664 10990 2012 1996 2168 2051 1006 2025 1037 26917 1007 1012 2023 2466 1999 1996 2398 1997 5365 2071 2031 2468 2074 102\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:label: 1 (id = 1)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:label: 1 (id = 1)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Example ***\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Example ***\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:guid: None\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:guid: None\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:tokens: [CLS] i loved this movie ! ! jack black and kyle gas ##s have probably made one of the best comedies since up in smoke . this movie has something for everyone : rock , drugs , and satan . what more could someone want ? i suggest this movie to anyone that wants to sit back and forget all the trouble in the world and have a good laugh . even if the movie does not do well in the theaters it is sure to become a cult classic . i do suggest anyone going to see the movie pick up the d ' s first album because a lot of the jokes are made only fun ##nier if you have heard the songs ( such [SEP]\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:tokens: [CLS] i loved this movie ! ! jack black and kyle gas ##s have probably made one of the best comedies since up in smoke . this movie has something for everyone : rock , drugs , and satan . what more could someone want ? i suggest this movie to anyone that wants to sit back and forget all the trouble in the world and have a good laugh . even if the movie does not do well in the theaters it is sure to become a cult classic . i do suggest anyone going to see the movie pick up the d ' s first album because a lot of the jokes are made only fun ##nier if you have heard the songs ( such [SEP]\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_ids: 101 1045 3866 2023 3185 999 999 2990 2304 1998 7648 3806 2015 2031 2763 2081 2028 1997 1996 2190 22092 2144 2039 1999 5610 1012 2023 3185 2038 2242 2005 3071 1024 2600 1010 5850 1010 1998 16795 1012 2054 2062 2071 2619 2215 1029 1045 6592 2023 3185 2000 3087 2008 4122 2000 4133 2067 1998 5293 2035 1996 4390 1999 1996 2088 1998 2031 1037 2204 4756 1012 2130 2065 1996 3185 2515 2025 2079 2092 1999 1996 12370 2009 2003 2469 2000 2468 1037 8754 4438 1012 1045 2079 6592 3087 2183 2000 2156 1996 3185 4060 2039 1996 1040 1005 1055 2034 2201 2138 1037 2843 1997 1996 13198 2024 2081 2069 4569 14862 2065 2017 2031 2657 1996 2774 1006 2107 102\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_ids: 101 1045 3866 2023 3185 999 999 2990 2304 1998 7648 3806 2015 2031 2763 2081 2028 1997 1996 2190 22092 2144 2039 1999 5610 1012 2023 3185 2038 2242 2005 3071 1024 2600 1010 5850 1010 1998 16795 1012 2054 2062 2071 2619 2215 1029 1045 6592 2023 3185 2000 3087 2008 4122 2000 4133 2067 1998 5293 2035 1996 4390 1999 1996 2088 1998 2031 1037 2204 4756 1012 2130 2065 1996 3185 2515 2025 2079 2092 1999 1996 12370 2009 2003 2469 2000 2468 1037 8754 4438 1012 1045 2079 6592 3087 2183 2000 2156 1996 3185 4060 2039 1996 1040 1005 1055 2034 2201 2138 1037 2843 1997 1996 13198 2024 2081 2069 4569 14862 2065 2017 2031 2657 1996 2774 1006 2107 102\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:label: 1 (id = 1)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:label: 1 (id = 1)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Example ***\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Example ***\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:guid: None\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:guid: None\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:tokens: [CLS] it ' s perfectly ok that people dies in an animation , but there are just way too many death in this one . start from the very beginning , the story is all around battles , fights , death , and revenge . it goes on and on for entire one and a half hour . it was interesting at the beginning , but i grew very tire after before the show was half way through . unlike other animation ##s , this one is lack of humor . there are not many interactions between the characters either . the good thing about it is the sword fight scene looks pretty good and the characters look nice . [SEP]\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:tokens: [CLS] it ' s perfectly ok that people dies in an animation , but there are just way too many death in this one . start from the very beginning , the story is all around battles , fights , death , and revenge . it goes on and on for entire one and a half hour . it was interesting at the beginning , but i grew very tire after before the show was half way through . unlike other animation ##s , this one is lack of humor . there are not many interactions between the characters either . the good thing about it is the sword fight scene looks pretty good and the characters look nice . [SEP]\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_ids: 101 2009 1005 1055 6669 7929 2008 2111 8289 1999 2019 7284 1010 2021 2045 2024 2074 2126 2205 2116 2331 1999 2023 2028 1012 2707 2013 1996 2200 2927 1010 1996 2466 2003 2035 2105 7465 1010 9590 1010 2331 1010 1998 7195 1012 2009 3632 2006 1998 2006 2005 2972 2028 1998 1037 2431 3178 1012 2009 2001 5875 2012 1996 2927 1010 2021 1045 3473 2200 12824 2044 2077 1996 2265 2001 2431 2126 2083 1012 4406 2060 7284 2015 1010 2023 2028 2003 3768 1997 8562 1012 2045 2024 2025 2116 10266 2090 1996 3494 2593 1012 1996 2204 2518 2055 2009 2003 1996 4690 2954 3496 3504 3492 2204 1998 1996 3494 2298 3835 1012 102 0 0 0 0 0 0 0\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_ids: 101 2009 1005 1055 6669 7929 2008 2111 8289 1999 2019 7284 1010 2021 2045 2024 2074 2126 2205 2116 2331 1999 2023 2028 1012 2707 2013 1996 2200 2927 1010 1996 2466 2003 2035 2105 7465 1010 9590 1010 2331 1010 1998 7195 1012 2009 3632 2006 1998 2006 2005 2972 2028 1998 1037 2431 3178 1012 2009 2001 5875 2012 1996 2927 1010 2021 1045 3473 2200 12824 2044 2077 1996 2265 2001 2431 2126 2083 1012 4406 2060 7284 2015 1010 2023 2028 2003 3768 1997 8562 1012 2045 2024 2025 2116 10266 2090 1996 3494 2593 1012 1996 2204 2518 2055 2009 2003 1996 4690 2954 3496 3504 3492 2204 1998 1996 3494 2298 3835 1012 102 0 0 0 0 0 0 0\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:label: 0 (id = 0)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:label: 0 (id = 0)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Example ***\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Example ***\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:guid: None\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:guid: None\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:tokens: [CLS] a rich experience is to be gained from watching this film . < br / > < br / > this is a seemingly simple story of a gifted pupil , yi ##u kw ##ok , who later becomes a teacher of classical chinese poetry . he is married with two sons , and things at home seem normal . he still loves his wife after twenty years of marriage , and his sons alternately fill him with pride ( the elder one ) and disappointment ( the younger one ) . < br / > < br / > his passion for the poetry makes him an object of in ##fat ##uation for a senior student in one of his classes . the student , [SEP]\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:tokens: [CLS] a rich experience is to be gained from watching this film . < br / > < br / > this is a seemingly simple story of a gifted pupil , yi ##u kw ##ok , who later becomes a teacher of classical chinese poetry . he is married with two sons , and things at home seem normal . he still loves his wife after twenty years of marriage , and his sons alternately fill him with pride ( the elder one ) and disappointment ( the younger one ) . < br / > < br / > his passion for the poetry makes him an object of in ##fat ##uation for a senior student in one of his classes . the student , [SEP]\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_ids: 101 1037 4138 3325 2003 2000 2022 4227 2013 3666 2023 2143 1012 1026 7987 1013 1028 1026 7987 1013 1028 2023 2003 1037 9428 3722 2466 1997 1037 12785 11136 1010 12316 2226 6448 6559 1010 2040 2101 4150 1037 3836 1997 4556 2822 4623 1012 2002 2003 2496 2007 2048 4124 1010 1998 2477 2012 2188 4025 3671 1012 2002 2145 7459 2010 2564 2044 3174 2086 1997 3510 1010 1998 2010 4124 23554 6039 2032 2007 6620 1006 1996 6422 2028 1007 1998 10520 1006 1996 3920 2028 1007 1012 1026 7987 1013 1028 1026 7987 1013 1028 2010 6896 2005 1996 4623 3084 2032 2019 4874 1997 1999 27753 14505 2005 1037 3026 3076 1999 2028 1997 2010 4280 1012 1996 3076 1010 102\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_ids: 101 1037 4138 3325 2003 2000 2022 4227 2013 3666 2023 2143 1012 1026 7987 1013 1028 1026 7987 1013 1028 2023 2003 1037 9428 3722 2466 1997 1037 12785 11136 1010 12316 2226 6448 6559 1010 2040 2101 4150 1037 3836 1997 4556 2822 4623 1012 2002 2003 2496 2007 2048 4124 1010 1998 2477 2012 2188 4025 3671 1012 2002 2145 7459 2010 2564 2044 3174 2086 1997 3510 1010 1998 2010 4124 23554 6039 2032 2007 6620 1006 1996 6422 2028 1007 1998 10520 1006 1996 3920 2028 1007 1012 1026 7987 1013 1028 1026 7987 1013 1028 2010 6896 2005 1996 4623 3084 2032 2019 4874 1997 1999 27753 14505 2005 1037 3026 3076 1999 2028 1997 2010 4280 1012 1996 3076 1010 102\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:label: 1 (id = 1)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:label: 1 (id = 1)\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Example ***\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:*** Example ***\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:guid: None\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:guid: None\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:tokens: [CLS] i had such high hopes for tel ##eto ##on retro to air this but instead of having shows such as this , ones that don ' t get the treatment that they deserve , they air things that i may have seen dozens of times before . < br / > < br / > the cent ##uri ##ons was the highlight of my pre - teen years . i know that may seem a little bit cl ##iche ##d but it ' s true . after duke from g . i . joe , jake rock ##ew ##ell is another one of those cartoon characters that i really had a crush on . < br / > < br / > it ' s too [SEP]\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:tokens: [CLS] i had such high hopes for tel ##eto ##on retro to air this but instead of having shows such as this , ones that don ' t get the treatment that they deserve , they air things that i may have seen dozens of times before . < br / > < br / > the cent ##uri ##ons was the highlight of my pre - teen years . i know that may seem a little bit cl ##iche ##d but it ' s true . after duke from g . i . joe , jake rock ##ew ##ell is another one of those cartoon characters that i really had a crush on . < br / > < br / > it ' s too [SEP]\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_ids: 101 1045 2018 2107 2152 8069 2005 10093 18903 2239 22307 2000 2250 2023 2021 2612 1997 2383 3065 2107 2004 2023 1010 3924 2008 2123 1005 1056 2131 1996 3949 2008 2027 10107 1010 2027 2250 2477 2008 1045 2089 2031 2464 9877 1997 2335 2077 1012 1026 7987 1013 1028 1026 7987 1013 1028 1996 9358 9496 5644 2001 1996 12944 1997 2026 3653 1011 9458 2086 1012 1045 2113 2008 2089 4025 1037 2210 2978 18856 17322 2094 2021 2009 1005 1055 2995 1012 2044 3804 2013 1043 1012 1045 1012 3533 1010 5180 2600 7974 5349 2003 2178 2028 1997 2216 9476 3494 2008 1045 2428 2018 1037 10188 2006 1012 1026 7987 1013 1028 1026 7987 1013 1028 2009 1005 1055 2205 102\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_ids: 101 1045 2018 2107 2152 8069 2005 10093 18903 2239 22307 2000 2250 2023 2021 2612 1997 2383 3065 2107 2004 2023 1010 3924 2008 2123 1005 1056 2131 1996 3949 2008 2027 10107 1010 2027 2250 2477 2008 1045 2089 2031 2464 9877 1997 2335 2077 1012 1026 7987 1013 1028 1026 7987 1013 1028 1996 9358 9496 5644 2001 1996 12944 1997 2026 3653 1011 9458 2086 1012 1045 2113 2008 2089 4025 1037 2210 2978 18856 17322 2094 2021 2009 1005 1055 2995 1012 2044 3804 2013 1043 1012 1045 1012 3533 1010 5180 2600 7974 5349 2003 2178 2028 1997 2216 9476 3494 2008 1045 2428 2018 1037 10188 2006 1012 1026 7987 1013 1028 1026 7987 1013 1028 2009 1005 1055 2205 102\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:label: 1 (id = 1)\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:label: 1 (id = 1)\n"
          ],
          "name": "stderr"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5N9jTDTkBOf7",
        "colab_type": "code",
        "outputId": "3d9f6c31-c09f-400d-e562-80b7cf928f72",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 88
        }
      },
      "source": [
        "print(type(train_features))\n",
        "print(type(test_features))\n",
        "print(train_features[0].__dict__)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "<class 'list'>\n",
            "<class 'list'>\n",
            "{'input_ids': [101, 2183, 2046, 2023, 3185, 1010, 1045, 2018, 2657, 2204, 2477, 2055, 2009, 1012, 2746, 2041, 1997, 2009, 1010, 1045, 2347, 1005, 1056, 2428, 15261, 4496, 9364, 1012, 4079, 25039, 2290, 3248, 1037, 2738, 24282, 2839, 2172, 2066, 2010, 2060, 5691, 1012, 2045, 2020, 1037, 3232, 1997, 11680, 2182, 1998, 2045, 1011, 1011, 2498, 2205, 6057, 1012, 2763, 2026, 5440, 3033, 1997, 1996, 3185, 2003, 2043, 2002, 11278, 1999, 1996, 2252, 3496, 1012, 1045, 6135, 10657, 3046, 2008, 2041, 2279, 2051, 1045, 2424, 2870, 1999, 1037, 2252, 1012, 1037, 3232, 1997, 3340, 2182, 1998, 2045, 2164, 1024, 12756, 4419, 1010, 11382, 19020, 24654, 3367, 1010, 2008, 14556, 2013, 1060, 1011, 6764, 1010, 1998, 5076, 7346, 1012, 1045, 2179, 2009, 3243, 19142, 2000, 2156, 1037, 12081, 3311, 1997, 102], 'input_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], 'segment_ids': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'label_id': 1, 'is_real_example': True}\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ccp5trMwRtmr",
        "colab_type": "text"
      },
      "source": [
        "#Creating a model\n",
        "\n",
        "Now that we've prepared our data, let's focus on building a model. `create_model` does just this below. First, it loads the BERT tf hub module again (this time to extract the computation graph). Next, it creates a single new layer that will be trained to adapt BERT to our sentiment task (i.e. classifying whether a movie review is positive or negative). This strategy of using a mostly trained model is called [fine-tuning](http://wiki.fast.ai/index.php/Fine_tuning)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6o2a5ZIvRcJq",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def create_model(is_predicting, input_ids, input_mask, segment_ids, labels,\n",
        "                 num_labels):\n",
        "  \"\"\"Creates a classification model.\"\"\"\n",
        "\n",
        "  bert_module = hub.Module(\n",
        "      BERT_MODEL_HUB,\n",
        "      trainable=True)\n",
        "  bert_inputs = dict(\n",
        "      input_ids=input_ids,\n",
        "      input_mask=input_mask,\n",
        "      segment_ids=segment_ids)\n",
        "  bert_outputs = bert_module(\n",
        "      inputs=bert_inputs,\n",
        "      signature=\"tokens\",\n",
        "      as_dict=True)\n",
        "\n",
        "  # Use \"pooled_output\" for classification tasks on an entire sentence.\n",
        "  # Use \"sequence_outputs\" for token-level output.\n",
        "  output_layer = bert_outputs[\"pooled_output\"]\n",
        "\n",
        "  hidden_size = output_layer.shape[-1].value\n",
        "\n",
        "  # Create our own layer to tune for politeness data.\n",
        "  output_weights = tf.get_variable(\n",
        "      \"output_weights\", [num_labels, hidden_size],\n",
        "      initializer=tf.truncated_normal_initializer(stddev=0.02))\n",
        "\n",
        "  output_bias = tf.get_variable(\n",
        "      \"output_bias\", [num_labels], initializer=tf.zeros_initializer())\n",
        "\n",
        "  with tf.variable_scope(\"loss\"):\n",
        "\n",
        "    # Dropout helps prevent overfitting\n",
        "    output_layer = tf.nn.dropout(output_layer, keep_prob=0.9)\n",
        "\n",
        "    logits = tf.matmul(output_layer, output_weights, transpose_b=True)\n",
        "    logits = tf.nn.bias_add(logits, output_bias)\n",
        "    log_probs = tf.nn.log_softmax(logits, axis=-1)\n",
        "\n",
        "    # Convert labels into one-hot encoding\n",
        "    one_hot_labels = tf.one_hot(labels, depth=num_labels, dtype=tf.float32)\n",
        "\n",
        "    predicted_labels = tf.squeeze(tf.argmax(log_probs, axis=-1, output_type=tf.int32))\n",
        "    # If we're predicting, we want predicted labels and the probabiltiies.\n",
        "    if is_predicting:\n",
        "      return (predicted_labels, log_probs)\n",
        "\n",
        "    # If we're train/eval, compute loss between predicted and actual label\n",
        "    per_example_loss = -tf.reduce_sum(one_hot_labels * log_probs, axis=-1)\n",
        "    loss = tf.reduce_mean(per_example_loss)\n",
        "    return (loss, predicted_labels, log_probs)\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qpE0ZIDOCQzE",
        "colab_type": "text"
      },
      "source": [
        "Next we'll wrap our model function in a `model_fn_builder` function that adapts our model to work for training, evaluation, and prediction."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FnH-AnOQ9KKW",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# model_fn_builder actually creates our model function\n",
        "# using the passed parameters for num_labels, learning_rate, etc.\n",
        "def model_fn_builder(num_labels, learning_rate, num_train_steps,\n",
        "                     num_warmup_steps):\n",
        "  \"\"\"Returns `model_fn` closure for TPUEstimator.\"\"\"\n",
        "  def model_fn(features, labels, mode, params):  # pylint: disable=unused-argument\n",
        "    \"\"\"The `model_fn` for TPUEstimator.\"\"\"\n",
        "\n",
        "    input_ids = features[\"input_ids\"]\n",
        "    input_mask = features[\"input_mask\"]\n",
        "    segment_ids = features[\"segment_ids\"]\n",
        "    label_ids = features[\"label_ids\"]\n",
        "\n",
        "    is_predicting = (mode == tf.estimator.ModeKeys.PREDICT)\n",
        "    \n",
        "    # TRAIN and EVAL\n",
        "    if not is_predicting:\n",
        "\n",
        "      (loss, predicted_labels, log_probs) = create_model(\n",
        "        is_predicting, input_ids, input_mask, segment_ids, label_ids, num_labels)\n",
        "\n",
        "      train_op = bert.optimization.create_optimizer(\n",
        "          loss, learning_rate, num_train_steps, num_warmup_steps, use_tpu=False)\n",
        "\n",
        "      # Calculate evaluation metrics. \n",
        "      def metric_fn(label_ids, predicted_labels):\n",
        "        accuracy = tf.metrics.accuracy(label_ids, predicted_labels)\n",
        "        f1_score = tf.contrib.metrics.f1_score(\n",
        "            label_ids,\n",
        "            predicted_labels)\n",
        "        auc = tf.metrics.auc(\n",
        "            label_ids,\n",
        "            predicted_labels)\n",
        "        recall = tf.metrics.recall(\n",
        "            label_ids,\n",
        "            predicted_labels)\n",
        "        precision = tf.metrics.precision(\n",
        "            label_ids,\n",
        "            predicted_labels) \n",
        "        true_pos = tf.metrics.true_positives(\n",
        "            label_ids,\n",
        "            predicted_labels)\n",
        "        true_neg = tf.metrics.true_negatives(\n",
        "            label_ids,\n",
        "            predicted_labels)   \n",
        "        false_pos = tf.metrics.false_positives(\n",
        "            label_ids,\n",
        "            predicted_labels)  \n",
        "        false_neg = tf.metrics.false_negatives(\n",
        "            label_ids,\n",
        "            predicted_labels)\n",
        "        return {\n",
        "            \"eval_accuracy\": accuracy,\n",
        "            \"f1_score\": f1_score,\n",
        "            \"auc\": auc,\n",
        "            \"precision\": precision,\n",
        "            \"recall\": recall,\n",
        "            \"true_positives\": true_pos,\n",
        "            \"true_negatives\": true_neg,\n",
        "            \"false_positives\": false_pos,\n",
        "            \"false_negatives\": false_neg\n",
        "        }\n",
        "\n",
        "      eval_metrics = metric_fn(label_ids, predicted_labels)\n",
        "\n",
        "      if mode == tf.estimator.ModeKeys.TRAIN:\n",
        "        return tf.estimator.EstimatorSpec(mode=mode,\n",
        "          loss=loss,\n",
        "          train_op=train_op)\n",
        "      else:\n",
        "          return tf.estimator.EstimatorSpec(mode=mode,\n",
        "            loss=loss,\n",
        "            eval_metric_ops=eval_metrics)\n",
        "    else:\n",
        "      (predicted_labels, log_probs) = create_model(\n",
        "        is_predicting, input_ids, input_mask, segment_ids, label_ids, num_labels)\n",
        "\n",
        "      predictions = {\n",
        "          'probabilities': log_probs,\n",
        "          'labels': predicted_labels\n",
        "      }\n",
        "      return tf.estimator.EstimatorSpec(mode, predictions=predictions)\n",
        "\n",
        "  # Return the actual model function in the closure\n",
        "  return model_fn\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OjwJ4bTeWXD8",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Compute train and warmup steps from batch size\n",
        "# These hyperparameters are copied from this colab notebook (https://colab.sandbox.google.com/github/tensorflow/tpu/blob/master/tools/colab/bert_finetuning_with_cloud_tpus.ipynb)\n",
        "BATCH_SIZE = 32\n",
        "LEARNING_RATE = 2e-5\n",
        "NUM_TRAIN_EPOCHS = 3.0\n",
        "# Warmup is a period of time where hte learning rate \n",
        "# is small and gradually increases--usually helps training.\n",
        "WARMUP_PROPORTION = 0.1\n",
        "# Model configs\n",
        "SAVE_CHECKPOINTS_STEPS = 500\n",
        "SAVE_SUMMARY_STEPS = 100"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "emHf9GhfWBZ_",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Compute # train and warmup steps from batch size\n",
        "num_train_steps = int(len(train_features) / BATCH_SIZE * NUM_TRAIN_EPOCHS)\n",
        "num_warmup_steps = int(num_train_steps * WARMUP_PROPORTION)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oEJldMr3WYZa",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Specify outpit directory and number of checkpoint steps to save\n",
        "run_config = tf.estimator.RunConfig(\n",
        "    model_dir=OUTPUT_DIR,\n",
        "    save_summary_steps=SAVE_SUMMARY_STEPS,\n",
        "    save_checkpoints_steps=SAVE_CHECKPOINTS_STEPS)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "q_WebpS1X97v",
        "colab_type": "code",
        "outputId": "1648932a-7391-49d3-8af7-52d514e226e8",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 156
        }
      },
      "source": [
        "model_fn = model_fn_builder(\n",
        "  num_labels=len(label_list),\n",
        "  learning_rate=LEARNING_RATE,\n",
        "  num_train_steps=num_train_steps,\n",
        "  num_warmup_steps=num_warmup_steps)\n",
        "\n",
        "estimator = tf.estimator.Estimator(\n",
        "  model_fn=model_fn,\n",
        "  config=run_config,\n",
        "  params={\"batch_size\": BATCH_SIZE})\n"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Using config: {'_model_dir': 'gs://bert-tfhub/aclImdb_v1', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': 500, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true\n",
            "graph_options {\n",
            "  rewrite_options {\n",
            "    meta_optimizer_iterations: ONE\n",
            "  }\n",
            "}\n",
            ", '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7fcedb507be0>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NOO3RfG1DYLo",
        "colab_type": "text"
      },
      "source": [
        "Next we create an input builder function that takes our training feature set (`train_features`) and produces a generator. This is a pretty standard design pattern for working with Tensorflow [Estimators](https://www.tensorflow.org/guide/estimators)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1Pv2bAlOX_-K",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Create an input function for training. drop_remainder = True for using TPUs.\n",
        "train_input_fn = bert.run_classifier.input_fn_builder(\n",
        "    features=train_features,\n",
        "    seq_length=MAX_SEQ_LENGTH,\n",
        "    is_training=True,\n",
        "    drop_remainder=False)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "t6Nukby2EB6-",
        "colab_type": "text"
      },
      "source": [
        "Now we train our model! For me, using a Colab notebook running on Google's GPUs, my training time was about 14 minutes."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nucD4gluYJmK",
        "colab_type": "code",
        "outputId": "5d728e72-4631-42bf-c48d-3f51d4b968ce",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 68
        }
      },
      "source": [
        "print(f'Beginning Training!')\n",
        "current_time = datetime.now()\n",
        "estimator.train(input_fn=train_input_fn, max_steps=num_train_steps)\n",
        "print(\"Training took time \", datetime.now() - current_time)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Beginning Training!\n",
            "INFO:tensorflow:Skipping training since max_steps has already saved.\n",
            "Training took time  0:00:00.759709\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CmbLTVniARy3",
        "colab_type": "text"
      },
      "source": [
        "Now let's use our test data to see how well our model did:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JIhejfpyJ8Bx",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "test_input_fn = run_classifier.input_fn_builder(\n",
        "    features=test_features,\n",
        "    seq_length=MAX_SEQ_LENGTH,\n",
        "    is_training=False,\n",
        "    drop_remainder=False)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PPVEXhNjYXC-",
        "colab_type": "code",
        "outputId": "dd5482cd-c558-465f-c854-ec11a0175316",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 445
        }
      },
      "source": [
        "estimator.evaluate(input_fn=test_input_fn, steps=None)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Calling model_fn.\n",
            "INFO:tensorflow:Saver not created because there are no variables in the graph to restore\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "stream",
          "text": [
            "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gradients_impl.py:110: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
            "  \"Converting sparse IndexedSlices to a dense Tensor of unknown shape. \"\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Done calling model_fn.\n",
            "INFO:tensorflow:Starting evaluation at 2019-02-12T21:04:20Z\n",
            "INFO:tensorflow:Graph was finalized.\n",
            "INFO:tensorflow:Restoring parameters from gs://bert-tfhub/aclImdb_v1/model.ckpt-468\n",
            "INFO:tensorflow:Running local_init_op.\n",
            "INFO:tensorflow:Done running local_init_op.\n",
            "INFO:tensorflow:Finished evaluation at 2019-02-12-21:06:05\n",
            "INFO:tensorflow:Saving dict for global step 468: auc = 0.86659324, eval_accuracy = 0.8664, f1_score = 0.8659711, false_negatives = 375.0, false_positives = 293.0, global_step = 468, loss = 0.51870537, precision = 0.880457, recall = 0.8519542, true_negatives = 2174.0, true_positives = 2158.0\n",
            "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 468: gs://bert-tfhub/aclImdb_v1/model.ckpt-468\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'auc': 0.86659324,\n",
              " 'eval_accuracy': 0.8664,\n",
              " 'f1_score': 0.8659711,\n",
              " 'false_negatives': 375.0,\n",
              " 'false_positives': 293.0,\n",
              " 'global_step': 468,\n",
              " 'loss': 0.51870537,\n",
              " 'precision': 0.880457,\n",
              " 'recall': 0.8519542,\n",
              " 'true_negatives': 2174.0,\n",
              " 'true_positives': 2158.0}"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 59
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ueKsULteiz1B",
        "colab_type": "text"
      },
      "source": [
        "Now let's write code to make predictions on new sentences:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OsrbTD2EJTVl",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def getPrediction(in_sentences):\n",
        "  labels = [\"Negative\", \"Positive\"]\n",
        "  input_examples = [run_classifier.InputExample(guid=\"\", text_a = x, text_b = None, label = 0) for x in in_sentences] # here, \"\" is just a dummy label\n",
        "  input_features = run_classifier.convert_examples_to_features(input_examples, label_list, MAX_SEQ_LENGTH, tokenizer)\n",
        "  predict_input_fn = run_classifier.input_fn_builder(features=input_features, seq_length=MAX_SEQ_LENGTH, is_training=False, drop_remainder=False)\n",
        "  predictions = estimator.predict(predict_input_fn)\n",
        "  return [(sentence, prediction['probabilities'], labels[prediction['labels']]) for sentence, prediction in zip(in_sentences, predictions)]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-thbodgih_VJ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "pred_sentences = [\n",
        "  \"That movie was absolutely awful\",\n",
        "  \"The acting was a bit lacking\",\n",
        "  \"The film was creative and surprising\",\n",
        "  \"Absolutely fantastic!\"\n",
        "]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QrZmvZySKQTm",
        "colab_type": "code",
        "outputId": "3891fafb-a460-4eb8-fa6c-335a5bbc10e5",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 649
        }
      },
      "source": [
        "predictions = getPrediction(pred_sentences)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "INFO:tensorflow:Writing example 0 of 4\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:guid: \n",
            "INFO:tensorflow:tokens: [CLS] that movie was absolutely awful [SEP]\n",
            "INFO:tensorflow:input_ids: 101 2008 3185 2001 7078 9643 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:label: 0 (id = 0)\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:guid: \n",
            "INFO:tensorflow:tokens: [CLS] the acting was a bit lacking [SEP]\n",
            "INFO:tensorflow:input_ids: 101 1996 3772 2001 1037 2978 11158 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:label: 0 (id = 0)\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:guid: \n",
            "INFO:tensorflow:tokens: [CLS] the film was creative and surprising [SEP]\n",
            "INFO:tensorflow:input_ids: 101 1996 2143 2001 5541 1998 11341 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:label: 0 (id = 0)\n",
            "INFO:tensorflow:*** Example ***\n",
            "INFO:tensorflow:guid: \n",
            "INFO:tensorflow:tokens: [CLS] absolutely fantastic ! [SEP]\n",
            "INFO:tensorflow:input_ids: 101 7078 10392 999 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:input_mask: 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
            "INFO:tensorflow:label: 0 (id = 0)\n",
            "INFO:tensorflow:Calling model_fn.\n",
            "INFO:tensorflow:Saver not created because there are no variables in the graph to restore\n",
            "INFO:tensorflow:Done calling model_fn.\n",
            "INFO:tensorflow:Graph was finalized.\n",
            "INFO:tensorflow:Restoring parameters from gs://bert-tfhub/aclImdb_v1/model.ckpt-468\n",
            "INFO:tensorflow:Running local_init_op.\n",
            "INFO:tensorflow:Done running local_init_op.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MXkRiEBUqN3n",
        "colab_type": "text"
      },
      "source": [
        "Voila! We have a sentiment classifier!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ERkTE8-7oQLZ",
        "colab_type": "code",
        "outputId": "26c33224-dc2c-4b3d-f7b4-ac3ef0a58b27",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 221
        }
      },
      "source": [
        "predictions"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[('That movie was absolutely awful',\n",
              "  array([-4.9142293e-03, -5.3180690e+00], dtype=float32),\n",
              "  'Negative'),\n",
              " ('The acting was a bit lacking',\n",
              "  array([-0.03325794, -3.4200459 ], dtype=float32),\n",
              "  'Negative'),\n",
              " ('The film was creative and surprising',\n",
              "  array([-5.3589125e+00, -4.7171740e-03], dtype=float32),\n",
              "  'Positive'),\n",
              " ('Absolutely fantastic!',\n",
              "  array([-5.0434084 , -0.00647258], dtype=float32),\n",
              "  'Positive')]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 73
        }
      ]
    }
  ]
}